Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network

Despite having impressive vision-language (VL) pretraining with BERT-based encoder for VL understanding, the of a universal encoder-decoder both understanding and generation remains challenging. The difficulty originates from inherently different peculiarities two disciplines, e.g., tasks capitalize on unrestricted message passing across modalities, while only employ visual-to-textual passing. In this paper, we start two-stream decoupled design structure, in which cross-modal decoder are involved to separately perform each type proxy tasks, simultaneous pretraining. Moreover, pretraining, dominant way is replace some input visual/word tokens mask enforce multi-modal encoder/decoder reconstruct original tokens, but no token when fine-tuning downstream tasks. As an alternative, propose primary scheduled sampling strategy that elegantly mitigates such discrepancy via two-pass manner. Extensive experiments demonstrate compelling generalizability our pretrained by four Source code available at https://github.com/YehLi/TDEN.